{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# xpath练习题：招聘信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "html_str = \"\"\"\n",
    "<body>\n",
    "    <div class=\"job\">\n",
    "        <div class=\"job_title\">\n",
    "            Python爬虫工程师\n",
    "        </div>\n",
    "\n",
    "        <div class=\"job_request\">\n",
    "            985 211 本科 硕士\n",
    "        </div>\n",
    "        <div class=\"salary\"><span>20</span>k/m,<span>14</span>薪</div>\n",
    "        <div class=\"job_describe\">\n",
    "            java python scrapy linux docker 分布式 javascript\n",
    "        </div>\n",
    "        <address>\n",
    "            北京\n",
    "        </address>\n",
    "\n",
    "    </div>\n",
    "    <hr>\n",
    "\n",
    "\n",
    "\n",
    "    <div class=\"job\">\n",
    "        <div class=\"job_title\">\n",
    "            高级爬虫工程师\n",
    "        </div>\n",
    "\n",
    "        <div class=\"job_request\">\n",
    "            5年工作经验\n",
    "        </div>\n",
    "        <div class=\"salary\"> <span>20</span> k/m,<span>15</span>薪</div>\n",
    "        <div class=\"job_describe\">\n",
    "            java 电商 分布式爬虫 实时数据抓取\n",
    "        </div>\n",
    "        <address>\n",
    "            北京\n",
    "        </address>\n",
    "\n",
    "    </div>\n",
    "    <hr>\n",
    "    <div class=\"job\">\n",
    "        <div class=\"job_title\">\n",
    "            爬虫工程师\n",
    "        </div>\n",
    "\n",
    "        <div class=\"job_request\">\n",
    "            硕士及以上学历，特别优秀可降低学历要求\n",
    "        </div>\n",
    "        <div class=\"salary\"><span>20</span>k/m</div>\n",
    "        <div class=\"job_describe\">\n",
    "            Mysql、Redis、Nosql，熟悉elt数据清洗流程及常用方法\n",
    "        </div>\n",
    "        <address>\n",
    "            上海\n",
    "        </address>\n",
    "\n",
    "    </div>\n",
    "    <hr>\n",
    "    <div class=\"job\">\n",
    "        <div class=\"job_title\">\n",
    "            高级爬虫工程师\n",
    "        </div>\n",
    "\n",
    "        <div class=\"job_request\">\n",
    "            本科及以上学历，985/211优先\n",
    "        </div>\n",
    "        <div class=\"salary\"><span>8-30</span>k/m</div>\n",
    "        <div class=\"job_describe\">\n",
    "            熟悉HTML、JS、正则表达式、Jsoup或jQuery、XPath等，从结构化和非结构化数据中解析数据；\n",
    "            有验证码破解、数据挖掘、数据仓库搭建经验者优先；\n",
    "        </div>\n",
    "        <address>\n",
    "            北京\n",
    "        </address>\n",
    "\n",
    "    </div>\n",
    "    <hr>\n",
    "\n",
    "    <div class=\"job\">\n",
    "        <div class=\"job_title\">\n",
    "            高级爬虫工程师\n",
    "        </div>\n",
    "\n",
    "        <div class=\"job_request\">\n",
    "            能力者高薪资，不限学历，有大牛指导\n",
    "        </div>\n",
    "        <div class=\"salary\"><span>10-50</span>k/m,<span>13</span>薪</div>\n",
    "        <div class=\"job_describe\">\n",
    "            1.精通C#或Python或C++或Go语言等爬虫方向开发和应用\n",
    "            2.具有爬虫系统框架设计能力，以及爬虫项目规划能力\n",
    "            3.有微信/微博/APP数据抓取优先\n",
    "        </div>\n",
    "        <address>\n",
    "            北京\n",
    "        </address>\n",
    "\n",
    "    </div>\n",
    "    <hr>\n",
    "    <div class=\"job\">\n",
    "        <div class=\"job_title\">\n",
    "            Python开发工程师\n",
    "        </div>\n",
    "\n",
    "        <div class=\"job_request\">\n",
    "            本科及以上学历，985/211优先\n",
    "        </div>\n",
    "        <div class=\"salary\"><span>10-50</span>k/m,<span>14</span>薪</div>\n",
    "        <div class=\"job_describe\">\n",
    "            django flask tornado flask\n",
    "            全日制本科学历\n",
    "            熟练掌握常用数据结构和算法，并能灵活运用\n",
    "        </div>\n",
    "        <address>\n",
    "            北京\n",
    "        </address>\n",
    "\n",
    "    </div>\n",
    "    <hr>\n",
    "    <div class=\"job\">\n",
    "        <div class=\"job_title\">\n",
    "            高级Python开发工程师\n",
    "        </div>\n",
    "\n",
    "        <div class=\"job_request\">\n",
    "            5-10年工作经验\n",
    "        </div>\n",
    "        <div class=\"salary\"><span>30</span>k/m,<span>15</span>薪</div>\n",
    "        <div class=\"job_describe\">\n",
    "            Python django devops restful postgresql docker ACM\n",
    "        </div>\n",
    "        <address>\n",
    "            上海\n",
    "        </address>\n",
    "\n",
    "    </div>\n",
    "    <hr>\n",
    "</body>\n",
    "\"\"\"\n",
    "from lxml import etree\n",
    "html = etree.HTML(html_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 列出所有的招聘信息标题，xpath越短越好"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7 ['\\n            Python爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            Python开发工程师\\n        ', '\\n            高级Python开发工程师\\n        ']\n"
     ]
    }
   ],
   "source": [
    "job_titles = html.xpath(\".//div[@class='job']/div[1]/text()\")\n",
    "print(len(job_titles),job_titles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分别统计python爬虫和python开发的岗位数，xpath越短越好"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 ['\\n            Python爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ']\n",
      "2 ['\\n            Python开发工程师\\n        ', '\\n            高级Python开发工程师\\n        ']\n"
     ]
    }
   ],
   "source": [
    "pc = html.xpath(\".//div[@class='job']/div[1][contains(text(),'爬虫')]/text()\")\n",
    "print(len(pc),pc)\n",
    "kf = html.xpath(\".//div[@class='job']/div[1][contains(text(),'Python开发')]/text()\")\n",
    "print(len(kf),kf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分别统计不同地区的岗位数，xpath越短越好"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 ['\\n            上海\\n        ', '\\n            上海\\n        ']\n",
      "5 ['\\n            北京\\n        ', '\\n            北京\\n        ', '\\n            北京\\n        ', '\\n            北京\\n        ', '\\n            北京\\n        ']\n"
     ]
    }
   ],
   "source": [
    "shs = html.xpath(\".//address[contains(text(),'上海')]/text()\")\n",
    "print(len(shs),shs)\n",
    "bjs = html.xpath(\".//address[contains(text(),'北京')]/text()\")\n",
    "print(len(bjs),bjs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查询有多少技术用到了docker技术，xpath越短越好"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 ['\\n            java python scrapy linux docker 分布式 javascript\\n        ', '\\n            Python django devops restful postgresql docker ACM\\n        ']\n"
     ]
    }
   ],
   "source": [
    "dockers = html.xpath(\".//div[@class='job_describe'][contains(text(),'docker')]/text()\")\n",
    "print(len(dockers),dockers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查询需要本科学历的招聘信息标题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 ['\\n            985 211 本科 硕士\\n        ', '\\n            本科及以上学历，985/211优先\\n        ', '\\n            本科及以上学历，985/211优先\\n        ']\n",
      "3 ['\\n            Python爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            Python开发工程师\\n        ']\n"
     ]
    }
   ],
   "source": [
    "bks = html.xpath(\".//div[@class='job_request'][contains(text(),'本科')]/text()\")\n",
    "print(len(bks),bks)\n",
    "\n",
    "bkts = html.xpath(\".//div[@class='job_request'][contains(text(),'本科')]/../div[1]/text()\")\n",
    "print(len(bkts),bkts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 写明薪资次数的岗位标题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 ['14', '15', '13', '14', '15']\n",
      "5 ['\\n            Python爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            Python开发工程师\\n        ', '\\n            高级Python开发工程师\\n        ']\n"
     ]
    }
   ],
   "source": [
    "salary = html.xpath('.//div[@class=\"salary\"]/span[2]/text()')\n",
    "print(len(salary),salary)\n",
    "salarys = html.xpath('.//div[@class=\"salary\"]/span[2]/../../div[1]/text()')\n",
    "print(len(salarys),salarys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 未写明自薪资次数的岗位标题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 ['\\n            爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ']\n"
     ]
    }
   ],
   "source": [
    "salarys = html.xpath(\".//div[@class='salary'][not(span[2])]/../div[1]/text()\")\n",
    "print(len(salarys),salarys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 具体薪资的岗位标题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 ['\\n            Python爬虫工程师\\n        ', '\\n            高级爬虫工程师\\n        ', '\\n            爬虫工程师\\n        ', '\\n            高级Python开发工程师\\n        ']\n"
     ]
    }
   ],
   "source": [
    "salarys = html.xpath(\".//div[@class='salary']/span[1][not(contains(text(),'-'))]/../../div[1]/text()\")\n",
    "print(len(salarys),salarys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 格式化输出全部的招聘信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n"
     ]
    }
   ],
   "source": [
    "jobs = html.xpath('.//div[@class=\"job\"]')\n",
    "print(len(jobs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\n            Python爬虫工程师\\n        ']\n",
      "['\\n            985 211 本科 硕士\\n        ']\n",
      "['20', 'k/m,', '14', '薪']\n",
      "['\\n            java python scrapy linux docker 分布式 javascript\\n        ']\n",
      "['\\n            北京\\n        ']\n",
      "-------------------------------------------------------\n",
      "['\\n            高级爬虫工程师\\n        ']\n",
      "['\\n            5年工作经验\\n        ']\n",
      "[' ', '20', ' k/m,', '15', '薪']\n",
      "['\\n            java 电商 分布式爬虫 实时数据抓取\\n        ']\n",
      "['\\n            北京\\n        ']\n",
      "-------------------------------------------------------\n",
      "['\\n            爬虫工程师\\n        ']\n",
      "['\\n            硕士及以上学历，特别优秀可降低学历要求\\n        ']\n",
      "['20', 'k/m']\n",
      "['\\n            Mysql、Redis、Nosql，熟悉elt数据清洗流程及常用方法\\n        ']\n",
      "['\\n            上海\\n        ']\n",
      "-------------------------------------------------------\n",
      "['\\n            高级爬虫工程师\\n        ']\n",
      "['\\n            本科及以上学历，985/211优先\\n        ']\n",
      "['8-30', 'k/m']\n",
      "['\\n            熟悉HTML、JS、正则表达式、Jsoup或jQuery、XPath等，从结构化和非结构化数据中解析数据；\\n            有验证码破解、数据挖掘、数据仓库搭建经验者优先；\\n        ']\n",
      "['\\n            北京\\n        ']\n",
      "-------------------------------------------------------\n",
      "['\\n            高级爬虫工程师\\n        ']\n",
      "['\\n            能力者高薪资，不限学历，有大牛指导\\n        ']\n",
      "['10-50', 'k/m,', '13', '薪']\n",
      "['\\n            1.精通C#或Python或C++或Go语言等爬虫方向开发和应用\\n            2.具有爬虫系统框架设计能力，以及爬虫项目规划能力\\n            3.有微信/微博/APP数据抓取优先\\n        ']\n",
      "['\\n            北京\\n        ']\n",
      "-------------------------------------------------------\n",
      "['\\n            Python开发工程师\\n        ']\n",
      "['\\n            本科及以上学历，985/211优先\\n        ']\n",
      "['10-50', 'k/m,', '14', '薪']\n",
      "['\\n            django flask tornado flask\\n            全日制本科学历\\n            熟练掌握常用数据结构和算法，并能灵活运用\\n        ']\n",
      "['\\n            北京\\n        ']\n",
      "-------------------------------------------------------\n",
      "['\\n            高级Python开发工程师\\n        ']\n",
      "['\\n            5-10年工作经验\\n        ']\n",
      "['30', 'k/m,', '15', '薪']\n",
      "['\\n            Python django devops restful postgresql docker ACM\\n        ']\n",
      "['\\n            上海\\n        ']\n",
      "-------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for job in jobs:\n",
    "    print(job.xpath(\".//div[1]/text()\")) # 加 . 表示当前位置\n",
    "    print(job.xpath(\".//div[2]/text()\"))\n",
    "    print(job.xpath(\".//div[3]//text()\"))\n",
    "    print(job.xpath(\".//div[4]/text()\"))\n",
    "    print(job.xpath(\".//address/text()\"))\n",
    "    print('-------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
